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Method, System and Computer Readable Medium for an Intelligent Search 
Workstation for Computer Assisted Interpretation of Medical Images 

CROSS-REFERENCE TO RELATED APPLICATIONS 

This application claims priority under 35 U.S.C. § 1 19(e) to United States provisional 
patent applications serial number 60/180,162, filed on February 4, 2000 and serial number 
60/207,401, filed on May 30, 2000, the entire contents of both of which are incorporated by 
reference herein. 

BACKGROUND OF THE INVENTION 
FIELD OF THE INVENTION 

The invention relates generally to the field of computer-aided diagnosis and image 
display workstations. It describes a method, system and computer readable medium that 
employs an intelligent search workstation for the computer assisted interpretation of medical 
images. Upon viewing an unknown medical case, the workstation shows both computer 
analysis output as well as images of cases with known diagnoses (e.g., malignant vs. benign) 
and similar computer-extracted features. The similarity index used in the search can be 
chosen by the radiologist to be based on a single feature, multiple features, or on the 
computer estimate of the likelihood of disease (such as malignancy in breast cancer). The 
present invention also generally relates to computerized techniques for automated analysis of 
digital images, for example, as disclosed in one or more of U.S. Patents U.S. Patents 
4,839,807; 4,841,555; 4,851,984; 4,875,165; 4,907,156; 4,918,534; 5,072,384; 5,133,020; 
5,150,292; 5,224,177; 5,289,374; 5,319,549; 5,343,390; 5,359,513; 5,452,367; 5,463,548; 
5,491,627; 5,537,485; 5,598,481; 5,622,171; 5,638,458; 5,657,362; 5,666,434; 5,673,332; 
5,668,888; 5,732,697; 5,740,268; 5,790,690; 5,832,103; 5,873,824; 5,881,124; 5,931,780; 
5,974,165; 5,982,915; 5,984,870; 5,987,345; and 6,01 1,862; as well as U.S. patent 
applications 08/173,935; 08/398,307 (PCT Publication WO 96/27846); 08/536,149; 
08/562,087; 08/900,188; 08/900,189; 08/900,191; 08/900,361; 08/979,623; 08/979,639; 
08/982,282; 09/027,468; 09/027,685; 09/028,518; 09/053,798; 09/092,004; 09/121,719; 
09/131,162; 09/141,535; 09/156,413; 09/298,852; and 09/471,088; PCT patent applications 
PCT/US99/24007; and PCT/US99/25998; and U.S. provisional patent applications 
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60/160,790; Attorney Docket Numbers 0730-0068-20 PROV and 0730-0069-20 PROV filed 
on January 18, 2000, all of which are incorporated herein by reference. 

The present invention includes use of various technologies referenced and described 
in the above-noted U.S. Patents and Applications, as well as described in the references 
identified in the appended LIST OF REFERENCES by the author(s) and year of publication 
and cross-referenced throughout the specification by numerals in brackets corresponding to 
the respective references, the entire contents of which, including the related patents and 
applications listed above and references listed in the LIST OF REFERENCES, are 
incorporated herein by reference. 

Discussion of the Background 

Breast cancer is a leading cause of death in women, causing an estimated 46,000 
deaths per year. Mammography is the most effective method for the early detection of breast 
cancer, and it has been shown that periodic screening of asymptomatic women does reduce 
mortality. Many breast cancers are detected and referred for surgical biopsy on the basis of a 
radiographically detected mass lesion or cluster of microcalcifications. Although general 
rules for the differentiation between benign and malignant mammographically identified 
breast lesions exist, considerable misclassification of lesions occurs with the current methods. 
On average, less than 30% of masses referred for surgical breast biopsy are actually 
malignant. 

Accordingly, due shortcomings in the above-noted methods, an improved method, 
system and computer readable medium for the computer assisted interpretation of medical 
images is desirable. 

SUMMARY OF THE INVENTION 

Accordingly, an object of this invention is to provide a method, system and computer 
readable medium that employs an intelligent search workstation for the computer assisted 
interpretation of medical images. 

Another object of the invention is to provide an automated method, system and 
computer readable medium that employs an intelligent search workstation for the computer 
assisted interpretation of medical images based on computer-estimated likelihood of a 
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pathological state, e.g., malignancy. 

Another object of the invention is to provide a method, system and computer readable 
medium that employs an intelligent search workstation for the computer assisted 
interpretation of medical images and outputs to the radiologist/physician output from the 
computer analysis of the medical images. 

These and other objects are achieved according to the invention by providing an 
improved method, system and computer readable medium for computer assisted 
interpretation of a medical image, including obtaining image data representative of a 
medical image; computing at least one feature characteristic of the image data; comparing 
the computed feature characteristic to corresponding computed feature characteristics 
derived from images in a known image data set; selecting image data from images of the 
known image data set having corresponding computed feature characteristics similar to the 
feature characteristics computed in the computing step; displaying at least one of the 
selected image data and the obtained image data. 

The present invention accordingly includes a computer readable medium storing 
program instructions by which the method of the invention can be performed when the stored 
program instructions are appropriately loaded into a computer, and a system for 
implementing the method of the invention. 

The method, system computer readable medium of the intelligent search workstation 
combines the benefit of computer-aided diagnosis with prior knowledge obtained via 
confirmed cases. It is expected that the display of known lesions with similar features will 
aid the radiologist in his/her work-up of a suspect lesion, especially when the radiologist's 
assessment of the lesion differs from the computer output, for the computerized assessment of 
tumor extent in magnetic resonance images. 

BRIEF DESCRIPTION OF THE DRAWINGS 
A more complete appreciation of the invention and many of the attendant advantages 
thereof will be readily obtained as the same becomes better understood by reference to the 
following detailed description when considered in connection with the accompanying 
drawings wherein: 

Figure 1 is a flow chart illustrating the overall scheme for a method that employs an 
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intelligent search workstation for the computer assisted interpretation of medical images; 

Figure 2 is a graph illustrating the distribution of cases of mammographic mass 
lesions in the known database; 

Figure 3 is a graph illustrating the distribution of tumor pathology in the database 
used in the demonstration and evaluation of the presented method, system and computer 
readable medium; 

Figure 4 is a flow chart illustrating the method for computer calculation of the 
features and estimate of the likelihood of malignancy; 

Figure 5 is a graph illustrating the performance in terms of an ROC curve of the 
computer analysis on the database used as the reference database in the intelligent search 
workstation, wherein the performance is given for the task of distinguishing malignant from 
benign lesions: 

Figure 6 is an image illustrating an example of the workstation interface for an 
unknown case using a color display where red indicates a malignant case and green indicates 
a benign case for a malignant case; 

Figure 7 is an image illustrating an example of the workstation interface for an 
unknown case using a color display where red indicates a malignant case and green indicates 
a benign case for a benign case; 

Figure 8 is a diagram illustrating the placement of the unknown case on the 
probability distribution axes of the malignant and benign cases, wherein instead of just 
displaying typical malignant and benign cases that are similar, the computer display shows 
relative similarity of the malignant and benign known cases by use of a color-coding of the 
similar lesions, whereby the probability distributions of the malignant and benign cases in the 
known database are shown by images along with the "location" of the unknown case relative 
to the two distributions; and 

Figure 9 is a schematic illustration of a general purpose computer which can be 
programmed according to the teachings of the present invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
The inventors are investigating the potential usefulness of computer-aided diagnosis 
as an aid to radiologists in the characterization and classification of mass lesions in 



-4- 



WO 01/57777 PCT/US01/00680 

mammography. Observer studies show that such a system can aid in increasing the 
diagnostic accuracy of radiologists both in terms of sensitivity and specificity. The present 
mass classification method, system and computer readable medium includes three 
components: (1) automated segmentation of mass regions, (2) automated feature-extraction, 
and (3) automated classification [1-3]. The method is initially trained with 95 mammograms 
containing masses from 65 patients. Features related to the margin, shape, and density of 
each mass are extracted automatically from the image data and merged into an estimate of the 
likelihood of malignancy using artificial neural networks (ANNs). These features include a 
spiculation measure, a radial gradient index, and two density measures. The round-robin 
performance of the computer in distinguishing between benign and malignant masses is 
evaluated by receiver operating characteristic (ROC) analysis. The computer classification 
scheme yielded an A z value of 0.94, similar to that of an experienced mammographer 
(A z =0.91) and statistically significantly higher than the average performance of five 
radiologists with less mammographic experience (A z =0.80). With the database used, the 
computer scheme achieved, at 100% sensitivity, a positive predictive value of 83%, which is 
12% higher than that of the experienced mammographer and 21% higher than that of the 
average performance of the less experienced mammographers at a p-value of less than 0.001 . 

The computerized mass classification method is next independently evaluated on a 
1 10-case clinical database consisting of 50 malignant and 60 benign cases. The effects of 
variations in both case mix and in film digitization technique on the performance of the 
method are assessed. Categorization of lesions as malignant or benign using the computer 
achieved an A z value (area under the ROC curve) of 0.90 on the prior training database (Fuji 
scanner digitization) in a round-robin evaluation, and A z values of 0.82 and 0.81 on the 
independent database for Konica and Lumisys digitization formats, respectively. However, 
the statistical comparison of these performances fails to show a statistical significant 
difference between the performance on the training database and that on the independent 
validation database (p-values > 0.10). Thus, the computer-based method for the classification 
of lesions on mammograms is shown to be robust to variations in case mix and film 
digitization technique. 

The inventors have now developed an intelligent search display into which has been 
incorporated the computerized mass classification method. Upon viewing an unknown 
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mammographic case, the display shows both the computer classification output as well as 
images of lesions with known diagnoses (e.g., malignant vs. benign) and similar computer- 
extracted features. The similarity index used in the search can be chosen by the radiologist to 
be based on a single feature, multiple features, or on the computer estimate of the likelihood 
of malignancy. 

The output of a computer-aided diagnostic scheme can take a variety of forms such as 
the estimated likelihood that a lesion is malignant either in terms of probabilities or along a 
standardized rating scale. This information is then available for use by the radiologist as he 
or she sees fit when making decisions regarding patient management. An alternative 
approach is for the computer to display a variety of lesions that have characteristics similar to 
the one at hand and for which the diagnosis is known, thereby providing a visual aid for the 
radiologist in decision making. Swett et al [4, 5] presented such a method in the 
Mammo/Icon system which used human input characteristics of the lesion in question. 
Sklansky et al [6] uses a visual neural network and displays the feature plot of the known 
database with the unknown case indicated. The intelligent workstation according to the 
present invention is unique in that it recalls lesions in the known database based either on a 
single feature, multiple features, or computer-estimate of the likelihood of malignancy. In 
addition, instead of just displaying typical malignant and benign cases that are similar, the 
computer display shows relative similarity of the malignant and benign known cases by use 
of a color-coding of the similar lesions. Basically, the probability distributions of the 
malignant and benign cases in the known database are shown by images along with the 
"location" of the unknown case relative to the two distributions. 

The intelligent search workstation combines the benefit of computer-aided diagnosis 
with prior knowledge obtained via confirmed cases. It is expected that the display of known 
lesions with similar features will aid the radiologist in his/her workup of a suspect lesion, 
especially when the radiologist's assessment of the lesion differs from the computer output. 

Referring now to the drawings, wherein like reference numerals designate identical 
or corresponding parts throughout the several views, and more particularly to Figure 1 
thereof, there is illustrated a top level block diagram of the method that employs an intelligent 
search workstation for the computer assisted interpretation of medical images according to 
the present invention. 
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The overall scheme inch des an initial acquisition of a set of known medical images 
that comprise a database, and presentation of the images in digital format. The lesion 
location in terms of estimated center is input from either a human or computer. The method, 
system and computer readable medium that employs an intelligent search workstation for the 
computer assisted interpretation of medical images consists of the following stages: access to 
a database of known medical images with known/confirmed diagnoses of pathological state 
(step 112), computer-extraction of features of lesions within the known database (step 104), 
input method for an unknown case (step 102), computer-extraction of features of lesion of the 
unknown case (step 104), calculation of likelihood of pathological state (e.g., likelihood of 
malignancy) for the known and unknown cases (step 108), calculation of similarity indices 
for the unknown cases relative to each of the known cases (step 1 10), search among the 
known database based on the calculated similarity indices (step 1 14) and presentation of the 
"similar" cases and/or the computer-estimated features and/or likelihood of pathological state 
(step 116). A specific example of the system is given for mass lesions in mammographic 
images in which the computer extracts features (step 104) and estimates the likelihood of 
malignancy for the known and the unknown cases (step 1 08), computes the similarity indices 
for each pair (step 1 10), and output cases that are similar in terms of individual features, 
combination of features, and/or computer-estimated likelihood of malignancy (step 1 16). The 
radiologist has the option of choosing various features, such as single feature, multiple 
features, likelihood of malignancy, etc., for the analysis (step 106). 

Database 

The images used are obtained by mammography followed by film digitization. The 
database used consists of 95 clinical mammograms (Kodak MinR screen/OM-1 film, 
Eastman Kodak, Rochester, NY), each containing a mass. Of the ninety five mammographic 
masses, 38 are benign and 57 are malignant. All but one case is verified by biopsy; the 
remaining one is deemed benign by long-term follow-up as illustrated in the graph of Figure 
2. The computerized mass classification method is independently evaluated on a 1 1 0-case 
clinical database consisting of 50 malignant and 60 benign cases as illustrated in the graph of 
Figure 3. 
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Automated Classification of Mass Lesions 

As shown in the flow chart of Figure 4, the mass classification method includes three 
components: (1) automated segmentation of mass regions including inputting of digital 
images (step 402), manual or automatic indication of lesion center (step 404), and automated 
lesion extraction (step 406), (2) automated feature-extraction (step 408) for predetermined 
features, such as spiculation, shape, margin sharpness, texture, etc. (step 410), and (3) 
automated classification (step 412). [1-3]. After automated determination of the likelihood of 
malignancy is preformed (step 414), the likelihood of malignancy and/or lesion features 
output (step 416) are input to the intelligent workstation (step 418). 

The segmentation (step 402) of a mass from the background parenchyma is 
accomplished using a multiple-transition-point, gray level region-growing technique. 
Segmentation begins within a 512 x 512 pixel region of interest manually or automatically 
centered about the abnormality in question (step 404). In clinical practice, the location of the 
mass can be identified either by a radiologist or by a computer-detection scheme and then fed 
into the classification scheme for an output on the likelihood of malignancy. In order to 
correct for the nonuniformity of the background distribution and to enhance image contrast 
for better segmentation of masses, background trend correction and histogram equalization 
techniques are applied to the 512 x 512 region of interest. 

The margin, shape and density of a mass are three major characteristics used by 
radiologists in classifying masses. Different characteristics of these features are associated 
with different levels of probabilities of malignancy. In order to determine the likelihood of 
malignancy associated with different margin and density characteristics, the inventors 
developed algorithms that extract (step 408), for example, two features that characterize the 
margin of a mass (e.g., spiculation and margin sharpness) and three features that characterize 
the density of a mass (e.g., average gray level, contrast and texture). 

Margin characteristics typically are very important discriminants in differentiating 
between benign and malignant masses. In order to determine the likelihood of malignancy of 
a mass based on its margin, two major margin characteristics - a spiculation measure and a 
margin-sharpness measure - are developed. Margin spiculation typically is the most 
important indicator for malignancy with spiculated lesions having a greater than 90% 
probability of malignancy. Margin sharpness typically is also very important in 
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determination of the benign vs. malignant nature of a mass -- with an ill-defined margin 
indicating possible malignancy and a well-defined margin indicating likely benignity. Only 
about 2% of well-defined masses are malignant. 

The spiculation measure is obtained from an analysis of radial edge gradients. The 
spiculation measure evaluates the average angle (degrees) by which the direction of the 
maximum gradient at each point along the margin of a mass deviates from the radial 
direction, the direction pointing from the geometric center of the mass to the point on the 
margin. The actual measure is the full width at half-maximum (FWHM) of the normalized 
edge-gradient distribution calculated for a neighborhood of the grown region of the mass with 
respect to the radial direction. This measure is able to quantify the degree of spiculation of a 
mass primarily because the direction of maximum gradient along the margin of a spiculated 
mass varies greatly from its radial direction, whereas the direction of the maximum gradient 
along the margin of a smooth mass is similar to its radial direction. 

The spiculation measure achieved a similar level of performance (A z =0.88) to that of 
the experienced mammographer's spiculation ratings (A z =0.85) in terms of the ability to 
distinguish between benign and malignant masses based solely on spiculation. [1] 

The sharpness of the margin of a mass, for example, can be described as well-defined, 
partially ill-defined or ill-defined. The average margin sharpness can be quantified by 
calculating the magnitude of the average gradient along the margin of the mass. A well- 
defined margin has a large value for the average margin sharpness measure, whereas an ill- 
defined margin has a small value. 

Although the radiographic density of a mass may not be by itself as powerful a 
predictor in distinguishing between benign and malignant masses as its margin features, taken 
with these features, density assessment typically can be extremely useful. The evaluation of 
the density of a mass is of particular importance in diagnosing circumscribed, lobulated, 
indistinct, or obscured masses that are not spiculated. 

In order to assess the density of a mass radiographically, introduced are three density- 
related measures (e.g., average gray level, contrast, and texture measure) that characterize 
different aspects of the density of a mass. These measures are similar to those used 
intuitively by radiologists. Average gray level is obtained by averaging the gray level values 
of each point within the grown region of a mass. Contrast is the difference between the 
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average gray level of the grown mass and the average gray level of the surrounding fatty 
areas (e.g., areas with gray level values in the lower 20% of the histogram for the total 
surrounding area). Texture is defined here as the standard deviation of the average gradient 
within a mass and it is used to quantify patterns arising from veins, trabeculae, and other 
structures that may be visible through a low-density mass, but not through a high-density 
mass. A mass of low radiographic density should have low values of average gray level and 
contrast, and a high value of the texture measure, whereas a mass of high radiographic 
density should have high values of average gray level and contrast, and a low value of the 
texture measure. 



Classifier 

Three automated classifiers (step 412) are investigated for the task of merging the 
computer-extracted features (step 408) into an estimate of the likelihood of malignancy (step 
414): (1) a rule-based method; (2) an artificial neural network; and (3) a hybrid system (i.e., 
combination of a one-step rule-based method and an artificial neural network). In 
determining the likelihood of malignancy (step 414) for the cases that had both the medio- 
lateral-oblique and cranio-caudal views, the measurements obtained from both views are 
considered and the one with the higher likelihood of malignancy estimated by the computer 
(step 414) is used in the evaluation. For example, in these cases, a mass would be classified 
as malignant if either one of the two views showed suspicious signs, e.g., either one of the 
FWHM measures from its two views satisfied the cutoff on the FWHM measure. 

A rule-based method adopts knowledge from experts into a set of simple rules. 
Certain criteria for differentiating between benign and malignant masses have been 
established by expert mammographers. The rules employed in the present approach for 
spiculation, margin-sharpness and density measures are based on these criteria. A two-step 
rule-based method is studied for this database. Because of its clinical diagnostic significance, 
the spiculation measure is applied first in the rule-based method. After the spiculation 
measure (e.g., FWHM) is applied to identify spiculated masses (e.g., including some irregular 
masses) and categorized them as malignant first, a second feature is applied to further 
characterize the masses in the non-spiculated category as previously discussed. In order to 
investigate the potential discriminant ability of the spiculation measure along with all the 
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possible secondary features, app ied separately is each of the remaining four features -- the 
margin-sharpness measure and f ie three density measures - after the spiculation measure. 
The threshold of the spiculation measure (e.g., FWHM of 160 degrees) is determined based 
on the entire database. The thresholds of the other four features are determined based on the 
remaining database only. 

The ANN approach is quite different from the rule-based method. Instead of using 
prespecified empirical algorithms based on prior knowledge, ANNs are able to learn from 
examples and therefore can acquire their own knowledge through learning. Also, neural 
networks are capable of processing large amounts of information simultaneously. Neural 
networks typically do not, however, provide the user with explanations about their decisions 
and may not be able to bring pre-existing knowledge into the network. Here employed is a 
conventional three-layer, feed-forward neural network with a back-propagation algorithm, 
which has been used in medical imaging and medical decision making. The structure of the 
neural network, for example, includes four input units (e.g., each of which corresponded to a 
computer-extracted feature), two hidden units, and one output unit. 

To determine the ability of the neural network to generalize from the training cases 
and make diagnoses for cases that had not been included in the database, employed is a 
round-robin method - also known as the leave-one-out method. In this method, all but one 
case is used to train the neural network. The single case that is left out is used to test the 
neural network. For the cases having both medio-lateral-oblique and cranio-caudal views, 
both images of the pair are left out in the round-robin training. The higher value of the two 
from the round-robin test is reported as the estimated likelihood of malignancy. This 
procedure is repeated for all the cases. 

Each classifier has its advantages and limitations. With rule-based methods, one can 
adopt pre-existing knowledge as rules. However, there are limitations in the availability of 
knowledge and knowledge translation. Even the experts find it difficult to articulate 
particular types of "intuitive" knowledge, and the process of translating particular knowledge 
into rules is limited by this expressive power. ANNs are capable of learning from examples 
and therefore can acquire their own knowledge. It may be most advantageous to use ANNs 
when intuitive knowledge cannot be explicitly expressed or is difficult to translate. However, 
the ANN requires a sufficiently large database to learn effectively. Also, with an ANN there 
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may be uncertainty as to whether the final learning goal is achieved in some situations. To 
take advantage of both rule-based systems and ANNs in the task of classifying masses, 
integrated is a rule-based method and an ANN into a hybrid system. In the hybrid system, 
initially is applied a rule on the spiculation measure since both spiculated and irregular 
masses are highly suspicious for malignancy, and then is applied an ANN to the remaining 
masses. Basically, this frees the ANN from having to "learn" the significance of spiculation 
to the detriment of learning the significance of the other features. 

The threshold of the spiculation measure for the hybrid system is the same as the one 
used in the rule-based method. The ANN applied in the hybrid system is a three-layer, feed- 
forward neural network with a back-propagation algorithm that has a structure of three input 
units (e.g.. corresponding to the three remaining features used in the ANN method), two 
hidden units, and one output unit. The same round-robin method is applied to test the 
generalization ability of this neural network to differentiate between benign and malignant 
masses in the non-spiculated category. 



Evaluation of Automated Classification of Mass Lesions 

The method is initially trained with 95 mammograms containing masses from 65 
patients. Features related to the margin, shape, and density of each mass are extracted 
automatically from the image data and merged into an estimate of the likelihood of 
malignancy using artificial neural networks. These features include a spiculation measure, a 
radial gradient index, and two density measures. The round-robin performance of the 
computer in distinguishing between benign and malignant masses is evaluated by receiver 
operating characteristic (ROC) analysis. As shown in Figure 5, the computer classification 
scheme based on the hybrid system yielded an A z value of 0.94, similar to that of an 
experienced mammographer (A z =0.91) and the computer classification scheme based on the 
ANN system (A z =0.90) and statistically significantly higher than the average performance of 
five radiologists with less mammographic experience (A 7 =0.80). With the database used, the 
computer scheme achieved, at 100% sensitivity, a positive predictive value of 83%, which is 
12% higher than that of the experienced mammographer and 21% higher than that of the 
average performance of the less experienced mammographers at a p- value of less than 0.001 . 
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The computerized mass classification method is next independently evaluated on a 
1 10-case clinical database consisting of 50 malignant and 60 benign cases. The effects of 
variations in both case mix and in film digitization technique on the performance of the 
method are assessed. Categorization of lesions as malignant or benign using the computer 
achieved an A z value (area under the ROC curve) of 0.90 on the prior training database (i.e., 
Fuji scanner digitization) in a round-robin evaluation, and A z values of 0.82 and 0.81 on the 
independent database for Konica and Lumisys digitization formats, respectively. However, 
the statistical comparison of these performances failed to show a statistical significant 
difference between the performance on the training database and that on the independent 
validation database (p-values > 0.10). Thus, the present computer-based method for the 
classification of lesions on mammograms is shown to be robust to variations in case mix and 
film digitization technique. 

Integration of Autom ated M a ss L esion Classification into the Intellip e nt Workstation 

The output of the automated classification method (Figure 4, step 416) is used to 
intelligently search for similar lesions in the known database (Figure 4, step 41 8 and Figure 1 
step 1 14). Figure 6 shows an example of the workstation graphical user interface (GUI) for 
an unknown case 602 using, for example, a color display where a red border 604 indicates a 
malignant case and green border 606 indicates a benign case for a case deemed malignant by 
the computer scheme. 

Figure 7 shows an example of the workstation GUI for an unknown case 702 using, 
for example, a color display where a red border 704 indicates a malignant case and a green 
border 706 indicates a benign case for a case deemed benign by the computer scheme. 
Similar displays can be given for black and white displays by, for example, outlining the 
known malignant lesions in black and outlining the known benign lesions in white. 

Referring to Figures 6 and 7, the user specifies how the search 608/708 is to occur - 
either by a single feature, such as likelihood of malignancy 608a/708a, spiculation 
608b/708b. radial gradient 608c/708c, margin sharpness 608d/708d, average gray level 
608e/708e, texture 608f/708f, etc., by a combination of features (in which, e.g., the 
Euclidean distance is calculated for the combination of features between the unknown case 
and each of the known cases), or by all of the features 608g/708g. The display in the Figures 
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6 and 7, for example, is a search by spiculation and under each known case the Euclidean 
distance measure as well as the spiculation measure 610/710 is given. The user has the 
option to see the computer outputs, the similar known cases, or both. 

The workstation interface shown in Figures 6 and 7 further includes a calculate button 
612/712, file/edit/select drop down menus 614/714, an open button 616/716, an exit button 
618/718, sharpness/brightness radio buttons 620/720, an information box 622/722 including 
yes/no buttons 622a/722a for showing similarities and CAD results, a display more images 
button 624/724 and a help drop down menu 626/726. 

Figure 8 further explains the search method by schematically showing the location of 
an unknown case 802 (e.g., which may be shown in blue) relative to the distribution of 
malignant 804 (e.g., which may be shown in red) and benign 806 (e.g., which may be shown 
in green) cases with respect to a given feature (and/or combination of features). The output 
of the cases can be in terms of those closest (in terms of absolute distance) as shown in 
Figures 6 and 7 or in terms of actual distance by not employing an absolute-type similarity 
index. For example, if only spiculation is chosen for searching, the unknown cases can be 
displayed with known cases on either side of it indicating which of the known cases are more 
spiculated and which ones are less spiculated. 

Observer Study Results 

At the Radiological Society of North America (RSNA) 2000, the inventors ran a 
demonstration of the intelligent search workstation of the present invention to show how it 
helps radiologists. The demo included 20 mammographic cases. For each case, the 
radiologists first saw the mammograms including the standard views of MLO, where MLO is 
medial-lateral oblique view (mammographic view of the breast at a side ways oblique angle) 
and CC, where CC is a cranial-caudal view (mammographic view of the breast from the head 
downwards), any special views of MLO and CC, as well as a 4 on 1 view showing the 
location of the lesion in the breast, where the 4 on 1 view is a collage of the left and right CC 
views on top of the left and right MLO views. Each radiologist was then asked to give 
his/her likelihood of malignancy on a scale from zero to 100, and also give his/her 
recommendation for either biopsy or follow-up. Then the computer output of the likelihood 
of malignancy was shown along with the resulting images from the intelligent search from 
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the reference library. Each radio ogist then was again asked for his/her likelihood of 
malignancy and recommendatior . After 20 cases, the sensitivity and specificity was 
calculated based on their biopsy/ bllow-up recommendation, where: 

Sensitivity is the percentage of malignant (i.e., cancerous) cases sent to biopsy. So a 
sensitivity of 0.92 is 92%. The higher the sensitivity the better. 

Specificity is determined by subtracting the false-positive rate FPR, i.e., the number 
of benign cases sent to biopsy, from unity, i.e, (1- FPR). The higher the specificity, the 
better. 

TABLE 1- RSNA 2000 Exhibit Observer Study Results on Mass Lesion Classification 
(Biopsy vs. Follow-Up) 



Reader Type 


Number of 
Observers 


Sensitivity 
(w/o CAD) 


Sensitivity 
(w/ CAD) 


Specificity 
(w/o CAD) 


Specificity 
(w/ CAD) 


Mammographers 


29 


0.81 


0.83 


0.62 


0.67 


Radiologist Non- 
Mammographers 


13 


0.79 


0.85 


0.55 


0.74 



In Table 1 above, "with CAD" includes the computer output of the likelihood of 
malignancy and the results of intelligent search on a known atlas of images, using the 
intelligent search workstation of the present invention. From Table 1, one sees that both the 
sensitivity and the specificity increased for mammographers and for radiologists (who do not 
read mammograms regularly) using the intelligent search workstation of the present 
invention. 

Figure 9 is a schematic illustration of a general purpose computer 900 which can be 
programmed according to the teachings of the present invention. In Figure 9, the computer 
900 implements the processes of the present invention, wherein the computer includes, for 
example, a display device 902 (e.g., a touch screen monitor with a touch-screen interface, 
etc.), a keyboard 904, a pointing device 906, a mouse pad or digitizing pad 908, a hard disk 
910, or other fixed, high density media drives, connected using an appropriate device bus 
(e.g., a SCSI bus, an Enhanced IDE bus, an Ultra DMA bus, a PCI bus, etc.), a floppy drive 
912, a tape or CD ROM drive 914 with tape or CD media 916, or other removable media 
devices, such as magneto-optical media, etc., and a mother board 918. The mother board 918 
includes, for example, a processor 920, a RAM 922, and a ROM 924 (e.g., DRAM, ROM, 
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EPROM, EEPROM, SRAM, SDRAM, and Flash RAM, etc.), I/O ports 926 which may be 
used to couple to an image acquisition device and optional special purpose logic devices (e.g., 
ASICs, etc.) or configurable logic devices (e.g., GAL and re-programmable FPGA) 928 for 
performing specialized hardware/software functions, such as sound processing, image 
processing, signal processing, neural network processing, automated classification, etc., a 
microphone 930, and a speaker or speakers 932. 

As stated above, the system includes at least one computer readable medium. 
Examples of computer readable media are compact discs, hard disks, floppy disks, tape, 
magneto-optical disks, PROMs (EPROM, EEPROM, Flash EPROM), DRAM, SRAM, 
SDRAM, etc. Stored on any one or on a combination of computer readable media, the 
present invention includes software for controlling both the hardware of the computer 900 
and for enabling the computer 900 to interact with a human user. Such software may include, 
but is not limited to, device drivers, operating systems and user applications, such as 
development tools. Such computer readable media further includes the computer program 
product of the present invention for performing any of the processes according to the present 
invention, described above. The computer code devices of the present invention can be any 
interpreted or executable code mechanism, including but not limited to scripts, interpreters, 
dynamic link libraries, Java classes, and complete executable programs, etc. 

The programming of general purpose computer 900 may include a software module 
for digitizing and storing images obtained from an image acquisition device. Alternatively, 
the present invention can also be implemented to process digital data derived from images 
obtained by other means, such as a picture archive communication system (PACS). In other 
words, often the digital images being processed will be in existence in digital form and need 
not be converted to digital form in practicing the invention. 

Accordingly, the mechanisms and processes set forth in the present description may 
be implemented using a conventional general purpose microprocessor or computer 
programmed according to the teachings in the present specification, as will be appreciated by 
those skilled in the relevant art(s). Appropriate software coding can readily be prepared by 
skilled programmers based on the teachings of the present disclosure, as will also be apparent 
to those skilled in the relevant art(s). However, as will be readily apparent to those skilled in 
the art, the present invention also may be implemented by the preparation of application- 
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specific integrated circuits or by interconnecting an appropriate network of conventional 
component circuits. 

The present invention thus also includes a computer-based product which may be 
hosted on a storage medium and include instructions which can be used to program a general 
purpose microprocessor or computer to perform processes in accordance with the present 
invention. This storage medium can include, but is not limited to, any type of disk including 
floppy disks, optical disks, CD-ROMs, magneto-optical disks, ROMs, RAMs, EPROMs, 
EEPROMs, flash memory, magnetic or optical cards, or any type of media suitable for 
storing electronic instructions. 

Recapitulating, the method, system and computer readable medium employs an 
intelligent search workstation for the computer assisted interpretation of medical images. The 
method, system and computer readable medium of the intelligent search interface takes as 
input an unknown medical image and searches through a set of cases with known diagnoses 
by way of a similarity index. The interface then displays images having similar features in a 
specified order. The search is directed by computer-extracted features (although it can easily 
be adapted for human-extracted features) and an estimate of the likelihood of the diagnosis of 
the unknown lesion is obtained with use of an artificial neural network (ANN, although 
others classifiers can be used). The similarity index is calculated using an Euclidean distance 
measure (although other measures of similarity can be employed). The displayed images can 
have their diagnostic truth displayed by means of enclosure by a green or red outline (for 
benign or malignant, respectively). In a sense, the display is showing the distribution of 
malignant and benign cases in the known database (or databank) that are similar to the 
unknown case. The significance of this method, system and computer readable medium is in 
the diagnosis of lesions and medical conditions, especially for confusing cases. The method, 
system and computer readable medium can also be used together with computerized output of 
the features and likelihood of malignancy (computer-aided diagnosis). 

Although the present invention is described in terms of practicing the method on 
mammographic image data sets, the intelligent workstation can be implemented for other 
medical images, such as chest radiography, ultrasound, magnetic resonance imaging, etc., in 
which a computerized analysis of image or lesion features is performed with respect to some 
disease state, as will be appreciated by those skilled in the relevant art(s). 
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Numerous modifications and variations of the present invention are possible in light 
of the above teachings. It is therefore to be understood that within the scope of the appended 
claims, the invention may be practiced otherwise than as specifically described herein. 
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Claims: 

1. A method for computer assisted interpretation of a medical image, comprising: 
obtaining image data representative of a medical image; 

computing at least one feature characteristic of the image data; 

comparing the computed feature characteristic to corresponding computed feature 
characteristics derived from images in a known image data set; 

selecting image data from images of the known image data set having corresponding 
computed feature characteristics similar to the feature characteristics computed in the 
computing step; and 

displaying at least one of the selected image data and the obtained image data. 

2. The method of Claim 1, further comprising: 

using a graphical user interface to display the image data. 

3. The method of Claim 1, further comprising: 

displaying a first indicator identifying which of the selected image data correspond 
to malignant lesions and a second indicator identifying which of the selected image data 
correspond to benign lesions. 

4. The method of Claim 3, further comprising: 

displaying as the first indicator a first colored border around the selected image data 
corresponding to malignant lesions and as the second indicator a second colored border 
around the selected image data corresponding to benign lesions. 

5. The method of Claim 1, further comprising: 

displaying the selected image data with or without an ASCII or numerical indicator 
corresponding to the computed feature characteristics derived from the images in the 
known image data set. 

6. The method of Claim 2, further comprising: 

displaying multiple views of the same obtained image data in the graphical user 
interface. 

7. The method of Claim 2, further comprising: 

using a selection mechanism in the graphical user interface to select at least one 
feature characteristic for use in the computing step. 

8. The method of Claim 2, further comprising: 
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using a region in the graphical user interface to display an enlarged view of the 
obtained image data relative to the displayed multiple views. 

9. The method of Claim 8, further comprising: 

using an adjustment mechanism in the graphical user interface to adjust sharpness 
and brightness of the enlarged view of the obtained image data. 

10. The method of Claim 5, further comprising: 

using a selection mechanism in the graphical user to selectively display the ASCII 
or numerical indicator. 

11. The method of Claim 1, wherein the computing step comprises: 
computing a margin characteristic feature including at least one of a spiculation 

measure and margin-sharpness measure. 

12. The method of Claim 11, wherein the computing step comprises: 
computing the spiculation measure by analysis of radial edge gradients, 

including evaluating an average angle by which a direction of a maximum gradient at each 
point along a margin of a mass within the image data deviates from a radial direction from 
a geometric center of the mass to a point on the margin to determine the spiculation 
measure. 

13. The method of Claim 12, wherein the computing step further comprises: 
calculating a normalized edge-gradient distribution for a neighborhood of a grown 

region of the mass with respect to the radial direction as the spiculation measure; and 

representing the computed spiculation measure by a full width at half-maximum 
(FWHM) of the calculated normalized edge-gradient distribution. 

14. The method of Claim 11, wherein the computing step comprises: 
calculating a magnitude of an average gradient along a margin of a mass within the 

image data to determine the margin sharpness measure. 

15. The method of Claim 1, wherein the computing step comprises: 
computing a density characteristic feature including at least one of average grey 

level, contrast and texture. 

16. The method of Claim 15, wherein the computing step comprises: 
averaging gray level values of each point within a grown region of a mass within 

the image data to determine the average grey level. 
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17. The method of Claim 15, wherein the computing step comprises: 
determining a difference between an average gray level of a grown mass within the 

image data and an average gray level of surrounding areas within the image data to 
determine the contrast. 

18. The method of Claim 15, wherein the computing step comprises: 
determining a standard deviation of an average gradient within a mass within the 

image data to determine the texture. 

19. The method of Claim 1, wherein: 

the comparing step comprises identifying plural retrieved known images defining a 
distribution of known malignant and benign abnormalities with respect to the computed 
feature characteristic; and 

the displaying step comprises displaying symbols representing a continuum of the 
known images ranging from malignant to benign abnormalities as a function of a value of 
the computed feature characteristic, and displaying a symbol representative of a position of 
candidate abnormalities in relation to the symbols of the continuum based on similarity of 
the computed feature characteristic and the corresponding feature characteristic existing in 
the known images. 

20. The method of Claim 1, wherein the computing step comprises: 
computing at least one feature characteristic from a group of feature characteristics 

consisting of spiculation, radial gradient, margin sharpness, average grey level, and 
texture. 

21 . The method of Claim 20, further comprising: 

applying the computed at least one feature characteristic to an artificial neural 
network; and 

determining a likelihood of malignancy based on an output unit of the artificial 
neural network. 

22. The method of Claim 21, wherein the applying step comprises: 
applying the computed at least one feature characteristic to the artificial neural 

network configured as a three-layered, feed-forward, backpropagation artificial neural 
network. 

23. The method of Claim 22, further comprising: 
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providing the three layers of the artificial neural network as plural input units, 
plural hidden units and one output unit, respectively. 

24. The method of Claim 23, wherein the applying step comprises: 

applying plural computed feature characteristics selected from the group consisting 
of the computed spiculation feature characteristic, the computed radial gradient feature 
characteristic, the computed margin sharpness feature characteristic, the computed average 
grey level feature characteristic, and the computed texture feature characteristic to the 
plural input units. 

25. The method of Claim 20, further comprising: 

applying the computed at least one feature characteristic to a rule-based classifier; 

and 

using the rule-based classifier to determine a likelihood of malignancy. 

26. The method of Claim 25, wherein the applying step comprises: 

applying the computed spiculation feature characteristic to the rule-based classifier. 

27. The method of Claim 26, wherein the applying step comprises: 
applying at least one computed feature characteristic selected from the group 

consisting of the computed radial gradient feature characteristic, the computed margin 
sharpness feature characteristic, the computed average grey level feature characteristic, and 
the computed texture feature characteristic to the rule-based classifier. 

28. The method of Claim 20, further comprising: 

applying the computed at least one feature characteristic to a rule-based classifier 
and an artificial neural network; and 

determining a likelihood of malignancy based on an output unit of the artificial 
neural network and using the rule-based classifier. 

29. The method of Claim 28, wherein the applying step comprises: 

applying the computed spiculation feature characteristic to the rule-based classifier. 

30. The method of Claim 29, wherein the applying step comprises: 
applying the computed at least one feature characteristic to the artificial neural 

network configured as a three-layered, feed-forward, backpropagation artificial neural 
network. 

31. The method of Claim 30, further comprising: 
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providing the three layers of the artificial neural network as plural input units, 
plural hidden units and one output unit, respectively. 

32. The method of Claim 31, wherein the applying step comprises: 

applying plural computed feature characteristics selected from the group consisting 
of the computed radial gradient feature characteristic, the computed margin sharpness 
feature characteristic, the computed average grey level feature characteristic, and the 
computed texture feature characteristic to the plural input units. 

33. A method for intelligent search of a known database for interpretation of 
medical images, comprising: 

generating image data from images derived from patients; 
computing feature characteristics of said image data; 

comparing said feature characteristics to computer-extracted feature characteristics 
of images in a known image data set; and 

displaying similar lesions with or without computer analysis output based on said 
comparison results. 

34. A method for computer-assisted interpretation of a medical image, comprising: 
obtaining digital image data representative of the medical image; 

processing the image data to extract at least one feature characterizing a candidate 
abnormality; 

searching a database of known image data corresponding to known images having 
said at least one feature and retrieving known image data having said at least one feature; 

comparing said extracted at least one feature with the at least one feature existing in 
the retrieved known image data to identify known retrieved images having said at least one 
feature similar to the extracted at least one feature; and 

displaying image data of a location of the medical image from which said extracted 
feature was extracted and known image data of a location in a retrieved known image 
having a similar feature. 

35. The method of Claim 34, wherein: 

said comparing step comprises identifying plural retrieved known images defining 
a distribution of known malignant and benign abnormalities with respect to said extracted 
feature; and 
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said displaying step comprises displaying symbols representing a continuum of said 
known images ranging from malignant to benign abnormalities as a function of a value of 
said at least one extracted feature, and displaying a symbol representative of a position of 
said candidate abnormality in relation to the symbols of said continuum based on similarity 
of the at least one extracted feature and the corresponding at least one feature existing in 
said known images. 

36. An image processing system configured to perform the steps recited in anyone 
of Claims 1 to 35. 

37. A storage medium storing a program for performing the steps recited in anyone 
of Claims 1 to 35. 
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